Image processing apparatus that corrects deterioration of image, image pickup apparatus, image processing method, and program

ABSTRACT

An image processing apparatus includes a data storing portion configured to store coefficient data for reconstructing an optical transfer function of an image pickup optical system in accordance with a type of the image pickup optical system and an imaging condition, a tap number determining portion configured to determine a tap number of the optical transfer function that is reconstructed by using the coefficient data in accordance with a size of one pixel of an image pickup element, and a reconstruction portion configured to reconstruct the optical transfer function in accordance with Nyquist frequency of the image pickup element and the tap number in a frequency space.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, and moreparticularly to an image processing apparatus that correctsdeterioration of an image caused by an image pickup optical system at ahigh resolution and at a high level of quality.

2. Description of the Related Art

With respect to an object taken via an image pickup optical system,light emitted from one point cannot be focused to one point and has asmall spread due to the influence of diffraction or aberration generatedby the image pickup optical system. The distribution having such a smallspread is called a point-spread function (PSF). Due to the influence ofthe image pickup optical system, a taken image is formed by convolvingthe PSF into an object image, and as a result the image is blurred andthe resolution is deteriorated.

Recently, it has been common that the taken image is stored aselectronic data, and a technology of correcting an image deteriorationcaused by an optical system using an image processing has been proposed.Japanese Patent Laid-Open No. 2010-56992 discloses a method of storing afilter coefficient for correcting the image deterioration to perform animage processing.

However, when the deterioration correction of the image is performed bya filter processing using an image restoration filter, it is necessarythat information of an optical transfer function (OTF information) formaking the image restoration filter are stored in an apparatus for eachpixel. However, since the OTF information is calculated by each of theinformation of an image pickup element and the image pickup opticalsystem, an amount of the information is significantly large and it isdifficult to store all of them in the apparatus.

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus thatreduces an amount of storage of data required for reconstructing anoptical transfer function of an image pickup optical system.

An image processing apparatus as one aspect of the present inventionincludes a data storing portion configured to store coefficient data forreconstructing an optical transfer function of an image pickup opticalsystem in accordance with a type of the image pickup optical system andan imaging condition, a tap number determining portion configured todetermine a tap number of the optical transfer function that isreconstructed by using the coefficient data in accordance with a size ofone pixel of an image pickup element, and a reconstruction portionconfigured to reconstruct the optical transfer function in accordancewith Nyquist frequency of the image pickup element and the tap number ina frequency space.

An image pickup apparatus as another aspect of the present inventionincludes an image pickup element, a data storing portion configured tostore coefficient data for reconstructing an optical transfer functionof an image pickup optical system in accordance with a type of the imagepickup optical system and an imaging condition, a tap number determiningportion configured to determine a tap number of the optical transferfunction that is reconstructed by using the coefficient data inaccordance with a size of a point-spread function of each image heightand a size of one pixel of an image pickup element, and a reconstructionportion configured to perform a sampling by the tap number up to Nyquistfrequency of the image pickup element in a frequency space toreconstruct the optical transfer function.

An image processing method as another aspect of the present inventionincludes the steps of selecting coefficient data for reconstructing anoptical transfer function of the image pickup optical system inaccordance with a type of the image pickup optical system and an imagingcondition, determining a tap number of the optical transfer functionthat is reconstructed by using the coefficient data in accordance with asize of a point-spread function of each image height and a size of onepixel of an image pickup element, reconstructing the optical transferfunction by performing a sampling by the tap number up to Nyquistfrequency of the image pickup element in a frequency space, making animage restoration filter for correcting an object image based on thereconstructed optical transfer function, and performing a filterprocessing for the object image on a real space using the imagerestoration filter.

A program as another aspect of the present invention is a program whichis configured so that a computer executes the steps of selectingcoefficient data for reconstructing an optical transfer function of theimage pickup optical system in accordance with a type of the imagepickup optical system and an imaging condition, determining a tap numberof the optical transfer function that is reconstructed by using thecoefficient data in accordance with a size of a point-spread function ofeach image height and a size of one pixel of an image pickup element,reconstructing the optical transfer function by performing a sampling bythe tap number up to Nyquist frequency of the image pickup element in afrequency space, making an image restoration filter for correcting anobject image based on the reconstructed optical transfer function, andperforming a filter processing for the object image on a real spaceusing the image restoration filter.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of an image processing apparatus inEmbodiment 1.

FIG. 2 is a diagram illustrating a coefficient calculating method thatis performed by a coefficient calculating apparatus in Embodiment 1.

FIG. 3 is a diagram illustrating output data of the coefficientcalculating apparatus in Embodiment 1.

FIG. 4 is a diagram illustrating a relationship between PSF and the tapnumber in Embodiment 1 (when the tap number is large).

FIG. 5 is a diagram illustrating a relationship between the PSF and thetap number in Embodiment 1 (when the tap number is small).

FIG. 6 is a diagram illustrating a method of making a reconstruction OTFby an OTF reconstruction portion in Embodiment 1.

FIG. 7 is a diagram illustrating reconstruction OTF parameters of theOTF reconstruction portion in Embodiment 1.

FIG. 8A is a diagram illustrating reconstruction OTF data of the OTFreconstruction portion in Embodiment 1.

FIG. 8B is a diagram illustrating reconstruction OTF data of the OTFreconstruction portion in Embodiment 1.

FIG. 9 is a diagram illustrating the tap number and PSF data inEmbodiment 1.

FIG. 10 is a conceptual diagram of an image restoration filter inEmbodiment 1.

FIG. 11 is a diagram illustrating an object image corrected by the imagerestoration filter in Embodiment 1.

FIG. 12 is a configuration diagram of an image pickup apparatusincluding an image processing apparatus in Embodiment 2.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

First of all, a method of performing a common image restorationprocessing will be described. When an image before receivingdeterioration caused by an optical system is defined as f(x, y), PSF(point-spread function) is defined as h(x, y), and a deteriorated imageis defined as g(x, y) on a real space (x, y), these are represented asfollowing Expression (1).g(x,y)=∫∫f(X,Y)*h(x−X,y−Y)dXdY  (1)

In Expression (1), f(x, y) denotes an image before receiving thedeterioration, g(x, y) denotes a deteriorated image, and h(x, y) denotesthe point-spread function (PSF). When the Fourier transform is performedfor Expression (1) described above to convert the real space (x, y) intoa frequency space (u, v), the relation represented by followingExpression (2) is met.G(u,v)=F(u,v)*H(u,v)  (2)

In Expression (2), F(u, v), G(u, v), and H(u, v) denote the Fouriertransforms of the f(x, y), g(x, y), and h(x, y), respectively.Therefore, following Expression (3) is met.F(u,v)=G(u,v)/H(u,v)  (3)

Expression (3) means that the Fourier transform F(u, v) of the imagebefore receiving the deterioration f(x, y) is obtained when the Fouriertransform G(u, v) of the deteriorated image g(x, y) is divided by theFourier transform H(u, v) of the point-spread function h(x, y) on thefrequency space. Accordingly, the image before receiving thedeterioration f(x, y) can be obtained by performing the inverse Fouriertransform for F(u, v).

However, if the image before receiving the deterioration is actuallyobtained by performing the processing, a noise generated by an imagepickup element is amplified and therefore a good image cannot beobtained.

As an image restoration method of suppressing the amplitude of thenoise, it is known that the Wiener filter represented by followingExpression (4) is used.1/H(u,v)*|H(u,v)|^2/(|H(u,v)^2+Γ)  (4)

In Expression (4), H(u, v) denotes an optical transfer function (OTF),and Γ denotes a constant which reduces an amplification amount of thenoise.

When the Wiener filter represented by Expression (4) described above ismultiplied by the OTF having the frequency and phase information of theimage pickup optical system, a phase of the PSF generated by adiffraction or an aberration of an optical system is set to be zero andfrequency characteristics are amplified to be able to obtain ahigh-resolution and high-quality image. In order to effectively useExpression (4), it is necessary to obtain accurate OTF information ofthe image pickup optical system. As a method of obtaining the OTFinformation, for example if there is design value information of theimage pickup optical system, the OTF information can be obtained by thecalculation based on the design value information. Alternatively, it canalso be obtained by imaging a point light source to perform the Fouriertransform for its intensity distribution. Generally, the opticalperformance, i.e. the F-number, the aberration, or the like, of theimage pickup optical system used in a camera varies in accordance withan image height. Therefore, in order to correct the deterioration of anobject image, the calculation on the frequency space cannot becollectively performed using Expression (4) in unchanged form, andtherefore Expression (4) is converted into a filter on the real spaceevery image height to perform the processing of correcting thedeterioration.

An optical image that is formed by the image pickup optical system iselectrically sampled by the image pickup element. Since the opticalimage that originally has a continuous quantity is converted into adiscrete value, the optical image becomes a frequency signal having acycle of a sampling frequency in the frequency space. Due to this cyclicproperty, when the frequency signal is distributed beyond one-half ofthe sampling frequency, the frequency signal is overlapped and anaccurate signal cannot be reproducible. The value of one-half of thesampling frequency is referred to as Nyquist frequency. The Nyquistfrequency is represented by fn=1/(2*b), where b is a size of one pixel(a pixel pitch) of the image pickup element.

The spatial frequency characteristics of the optical image immediatelybefore the image pickup element are represented by the OTF of the imagepickup optical system. When the image restoration processing isperformed, a size of one tap of the image restoration filter needs to beequal to a size of the image pickup element and preferably aperturecharacteristics of the image pickup element are also reflected. Theformer corresponds to performing the cutout of the OTF by the spatialfrequency in which the Nyquist frequency of the image pickup element isthe maximum value, and the latter corresponds to applying a low-passfilter to the OTF by the image pickup element. Therefore, the OTFinformation used for making the image restoration filter is not uniquelydetermined only by the image pickup optical system, and it also dependson the image pickup element.

Embodiment 1

First of all, an image processing apparatus in Embodiment 1 of thepresent invention will be described. FIG. 1 is a configuration diagramof the image processing apparatus as one example in the presentembodiment. The image processing apparatus of the present embodiment hasa function of correcting a deteriorated image, and it is for exampleconfigured as a program (software) which is configured so that acomputer executes the following image processing method.

In FIG. 1, the image processing apparatus includes a coefficientcalculating apparatus 100 (a coefficient calculating portion) thatcalculates a coefficient for reconstructing an optical transfer function(hereinafter, referred to also as an “OTF”). The coefficient calculatingapparatus 100 calculates the OTF based on a design value or a measuredvalue of the image pickup optical system (an image pickup lens 112). Thecoefficient calculating apparatus 100 (an order determining portion)converts the OTF into the coefficient (coefficient data), and determinesan order of the coefficient which is subsequently used for thereconstruction of the OTF in accordance with its accuracy (the accuracyof the fitting processing). Furthermore, the coefficient calculatingapparatus 100 determines the tap number required in reproducing the OTFsubsequently based on a size of the spatial distribution of thepoint-spread function (hereinafter, referred to also as a “PSF”). Thecoefficient calculating apparatus 100 calculates the coefficients up tothe determined order and the tap number of the OTF with respect to thecombination of various types of the image pickup optical systems (theimage pickup lenses 112) and the image pickup element 111 to output themto an image restoration processing apparatus 120.

A camera 110 includes the image pickup element 111 and the image pickuplens 112. The camera 110 adds an ID number (a lens ID) that specifiesthe image pickup lens 112, an imaging condition such as a stop, a zoom,or an object distance, and a value of the Nyquist frequency of thespatial frequency which can be expressed by the image pickup element 111to the object image which has been taken by the image pickup lens 112 tobe outputted.

The image restoration processing apparatus 120 stores various types ofinformation outputted from the coefficient calculating apparatus 100 andthe camera 110, and corrects the object image (the deteriorated image)obtained via the image pickup lens 112 using the information.Hereinafter, an internal configuration of the image restorationprocessing apparatus 120 will be described in detail. An imagerestoration information storing portion 121 stores the coefficient (thecoefficient data), the tap number, the lens ID, the imaging condition,and the Nyquist frequency to store them with respect to each of thecombinations of the various types of image pickup lens 112 and the imagepickup element 111 calculated by the coefficient calculating apparatus100. Thus, the image restoration information storing portion 121 is adata storing portion that stores the coefficient data for reconstructingthe OTF of the image pickup lens 112 or the like in accordance with thetype of the image pickup lens 112, the imaging condition, and the like.

An OTF reconstruction portion 122 (a reconstruction portion) obtains thelens ID of the image pickup lens 112, the imaging condition, and theNyquist frequency of the image pickup element 111 from the camera 110.The OTF reconstruction portion 122 also selects specific coefficient andtap number of the coefficients and the tap numbers which are stored inthe image restoration information storing portion 121 based on the lensID of the camera 110 used for taking the object image by a user and theimaging condition. Thus, the OTF reconstruction portion 122 is a tapnumber determining portion that determines the tap number of the OTFreconstructed using the coefficient in accordance with a size of the PSFof each image height and a size of one pixel of the image pickup element111. Even when the tap number is not determined in accordance with thesize of the PSF of each image height, the effect of the presentembodiment can be obtained by having a coefficient for which a fittingby a function has been performed. It is preferred that the tap number bedetermined in accordance with the size of the PSF of each image heightsince an extra value does not need to be included.

Furthermore, the OTF reconstruction portion 122 performs the samplingfrequencies up to the Nyquist frequency of the image pickup element 111in the frequency space to reconstruct the OTF used in a filterprocessing portion 123 using the selected coefficient and tap number.Thus, the OTF reconstruction portion 122 reconstructs the OTF inaccordance with the Nyquist frequency of the image pickup element 111and the tap number. Hereinafter, the OTF made by the OTF reconstructionportion 122 is also referred to as a reconstruction OTF. The filterprocessing portion 123 makes an image restoration filter for correctingthe deterioration of the taken object image using the reconstruction OTFmade by the OTF reconstruction portion 122. Then, the filter processingis performed for the object image on the real space using the imagerestoration filter to correct the object image.

Next, a coefficient calculating method in the present embodiment will bedescribed in detail. In the present embodiment, the OTF (the designvalue or the measured value) of the image pickup optical system (theimage pickup lens 112) is approximated by performing the fittingprocessing to a predetermined function to make the coefficient. As afunction used in the fitting processing, Legendre polynomial is used inthe present embodiment. However, the present embodiment is not limitedto this, and for example other expressions such as Chebushev polynomialmay also be used. The Legendre polynomial is represented as followingExpression (5). In Expression (5), [x] denotes the maximum integer whichis not more than x.

$\begin{matrix}{{P_{n}(x)} = {\frac{1}{2^{n}}{\sum\limits_{K = 0}^{\lbrack\frac{n}{2}\rbrack}{\left( {- 1} \right)^{k}\frac{\left( {{2\; n} - {2\; k}} \right)!}{{k!}{\left( {n - k} \right)!}{\left( {n - {2\; k}} \right)!}}x^{n - {2\; k}}}}}} & (5)\end{matrix}$

Since the OTF is represented by a form of z=f (x, y), a coefficienta_(ij) in following Expression (6) needs to be calculated in the presentembodiment.

$\begin{matrix}{z = {\sum\limits_{i}^{i = m}{\sum\limits_{j}^{j = n}{a_{iJ}{P(x)}_{i}{P(y)}_{j}}}}} & (6)\end{matrix}$

Expression (6) described above is an orthogonal function, and the valueof the coefficient a_(ij) is determined independently of the order inthe fitting processing. Thus, since the OTF of the image pickup opticalsystem is approximated by the fitting processing to the predeterminedfunction to make the coefficient, an amount of storage of necessary datacan be reduced. Using the characteristics of the orthogonal functionrepresented by Expression (6), if the fitting processing of the OTF canbe performed with sufficiently high accuracy only by using a low order,the processing can be terminated at this order and an amount ofinformation of the coefficient that is to be stored in the apparatus canbe suppressed.

FIG. 2 is a diagram illustrating a coefficient calculating method thatis performed by the coefficient calculating apparatus 100, andillustrates a specific method of performing the fitting processing ofthe optical transfer function (OTF) using Expressions (5) and (6)described above. In FIG. 2, reference signs fum and fvm are Nyquistfrequencies in meridional and sagittal directions of the OTF,respectively. Reference signs Nx and Ny are odd tap numbers in themeridional and sagittal directions of the OTF, respectively. Thecoefficient calculating apparatus 100 calculates the coefficient foreach of the real part and imaginary part of the OTF by performing thefitting processing.

The real part of the OTF is symmetrical in both the meridional directionand the sagittal direction. The imaginary part of the OTF is inverselysymmetrical in the meridional direction, and it is symmetrical in thesagittal direction. Due to the symmetry, data of the OTF for which thefitting are to be performed are sufficient if there are information ofareas having the symmetry such as at least ¼ area of whole of a definedarea. In the present embodiment, for the reason above, the ¼ area ofwhole of the defined area for both the real part and the imaginary partis cut out so as to contain a DC component to perform the fittingprocessing of the OTF with high accuracy. The present embodimentindicates an example of a case in which the OTF data are Nx (row)×Ny(column) taps, and data of 1 to [Nx/2]+1 rows and 1 to [Ny/2]+1 columnsare cut out, but the present embodiment is not limited to this.

FIG. 3 is one example of the output data of the coefficient calculatingapparatus 100, and illustrates the coefficients calculated by thecoefficient calculating method described above. As illustrated in FIG.3, in the present embodiment, the coefficients of the real part and theimaginary part of the OTF are calculated up to 10th order for both x andy each image height (image heights 1 to 10). The coefficients for eachimage height are collected and further the information of the lens ID,the stop, and the zoom, and the object distance are added to completeone coefficient data. In the present embodiment, as one example,coefficients for 10 image heights on conditions that the lens ID is “No.123”, the stop is “F2.8”, the zoom is “WIDE”, and the object distance is“close range” are illustrated. The coefficients made as illustrated inFIG. 3 may also be defined as a function for each order and each imageheight. The coefficient calculating apparatus 100 makes the informationfor all the combinations of the lens ID, the stop, the zoom, and theobject distance to be outputted.

Subsequently, a method of determining the tap number of thereconstruction OTF in the present embodiment will be described indetail. When the filter processing is performed for an image, theprocessing time significantly depends on the tap number of the filter.Therefore, it is preferred that the tap number of the filter be small ifa desired effect of correcting the image deterioration is obtained andany negative effect such as ringing does not occur during the filterprocessing.

The image restoration filter used for correcting the deteriorated imageis a filter on the real space. Accordingly, the tap number required forthe filter on the real space may be determined. Since the imagerestoration filter is a filter that corrects the deterioration of theimage caused by the point-spread function (PSF), an area similar to anarea where the PSF is distributed on the real space only has to beensured. In other words, the tap number required for the imagerestoration filter is the tap number of the PSF distribution area in thereal space. Since the real space and the frequency space have therelationship of the inverse each other, the tap number determined in thereal space can be used in the frequency space.

Referring to FIGS. 4 and 5, this will be described in detail. FIG. 4illustrates a case in which the tap number is applied to a sufficientlylarge area compared to the spatial distribution of the PSF. FIG. 5illustrates a case in which the tap number is applied to an areasubstantially the same as the spatial distribution of the PSF for thesame PSF as that of FIG. 4. In FIG. 4, the tap number in the real spacecorresponds to the minimum frequency pitch in the frequency space. Onthe other hand, reducing the tap number in the real space as illustratedin FIG. 5 means that the frequency space is roughly sampled, andindicates that the minimum frequency pitch is enlarged. In this case,the value of the Nyquist frequency in the frequency space does notchange. Therefore, when the tap number is reduced to cut the spatialdistribution of the PSF in the real space, the negative effect such asringing described above is easily generated during the image restorationprocessing. Accordingly, it is preferred that the tap number of thefilter be determined to be substantially the same area size as thespatial distribution of the PSF.

The image restoration information storing portion 121 stores thecoefficient (the coefficient data), the tap number, the lens ID, theimaging condition, and the Nyquist frequency outputted from thecoefficient calculating apparatus 100. The OTF reconstruction portion122 obtains the lens ID, the imaging condition, and the Nyquistfrequency from the camera 110. Subsequently, the OTF reconstructionportion 122 reads information of the selected tap number, lens ID,imaging condition, and Nyquist frequency from the image restorationinformation storing portion 121 to make the reconstruction OTF used formaking the image restoration filter using the information.

Next, referring to FIG. 6, a method of making the reconstruction OTF bythe OTF reconstruction portion 122 will be described in detail. TheNyquist frequencies in the meridional and sagittal directions requiredfor making the reconstruction OTF are defined as fuc_rm and fvc_im,respectively. The tap numbers in the meridional and sagittal directionsare defined as Mx and My, respectively. In the embodiment, with respectto the Nyquist frequencies fum and fvm, 0<fum_n≦fum, 0<fvm_n≦fvm,0<Mx≦Nx, and 0<My≦Ny are met, where the tap numbers Mx and My are oddnumbers.

Furthermore, x and y in Expressions (1) and (2) described above arereplaced with u and v, respectively, and areas of −fum_n/fum≦u≦1 and−fvm_n/fvm≦v≦1 are sampled by [Mx/2]+1 and [My/2]+1 taps, respectively.When the coefficient is substituted into Expression (2), the ¼ area (thequarter area) of the reconstruction OTF is made. The procedure issimilarly performed for both the real part (122-1-1) and the imaginarypart (122-2-1) of the reconstruction OTF. In the present embodiment, amethod of making the reconstruction OTF which has the defined area(domain) of −fum_n/fum≦u≦fum_n/fum and −fvm_n/fvm≦v≦fvm_n/fvm and thetap numbers Mx and My based on the reconstruction OTF of the ¼ area ofwhole of the defined area for both the real part and the imaginary partwill be described.

First of all, a method of making the real part of the reconstruction OTFwill be described. Using the real part (122-1-1) of the reconstructionOTF, the area is divided into an area of 1 to [Mx/2]+1 rows and 1 to[My/2] columns and an area of 1 to [Mx/2]+1 rows and [My/2]+1 column.Subsequently, numerical data of the area of 1 to [Mx/2]+1 rows and 1 to[My/2] columns are substituted into an area of 1 to [Mx/2]+1 rows and[My/2]+2 to My columns (the real part (122-1-2)) so as to beaxisymmetric with respect to the area of 1 to [Mx/2]+1 rows and [My/2]+1column.

Furthermore, the reconstruction OTF of the ½ area (half area) made bythe real part (122-1-2) is divided into an area of 1 to [Mx/2] rows and1 to My columns and an area of [Mx/2]+1 row and 1 to My columns (thereal part (122-1-3)). The numerical data of the area of 1 to [Mx/2] rowsand 1 to My columns are substituted into the area of [Mx/2]+2 row and 1to My columns so as to be axisymmetric with respect to the area of[Mx/2]+1 row and 1 to My columns.

Next, a method of making the imaginary part of the reconstruction OTFwill be described. The imaginary part (122-2-2) can be made by a methodsimilar to the real part (122-1-2). With respect to the imaginary part(122-2-3), the data need to be substituted by replacing positive andnegative signs. The method of making the imaginary part as describedabove is possible because of the characteristics of the OTF.

FIG. 7 illustrates a reconstruction OTF parameter of the OTFreconstruction portion 122, and is a relation diagram of the Nyquistfrequency and the tap number of the reconstruction OTF (across-sectional view of the reconstruction OTF). As described above, theNyquist frequency is a parameter which depends on a camera body (thecamera 110) determined based on the spatial resolution of the imagepickup element 111. The tap number is a parameter which depends on thePSF of the image pickup lens 112. Based on these two parameters and thecoefficient described above, a desired reconstruction OTF is made. InFIG. 7, with respect to the Nyquist frequency, f_nyq1>f_nyq2 is met, andwith respect to the tap number, N>M1>M2 is met. In the presentembodiment, as illustrated in FIG. 7, the Nyquist frequency and the tapnumber can be controlled so as to be desired values.

One example of a case in which the image processing for correcting thedeterioration of an object image has been performed by the above methodwill be described below. FIG. 8A illustrates the real part of thereconstruction OTF data of the OTF reconstruction portion 122, and FIG.8B illustrates the imaginary part of the reconstruction OTF data. InFIGS. 8A and 8B, reference numeral 201 denotes a reconstruction OTFwhich is made based on 10th order coefficient of the Legendre polynomialusing the OTF calculated from the design value information of the imagepickup optical system on condition that the Nyquist frequency is 238lines/mm and that the tap number is 193 on each side. Reference numerals202 and 203 denote reconstruction OTF which are obtained when the tapnumber is only changed without changing the Nyquist frequency. Referencenumeral 204 denotes a reconstruction OTF when the Nyquist frequency andthe tap number are changed at the same time. Thus, in the presentembodiment, the Nyquist frequency and the tap number can be controlled.Therefore, even when the combination of the image pickup lens 112 andthe camera body is changed, the present embodiment can be appliedwithout increasing stored data.

FIG. 9 is a diagram illustrating a relationship between the tap numberand the PSF data in the present embodiment. FIG. 9 illustrates the PSFcorresponding to FIGS. 8A and 8B, and the Nyquist frequency and the tapnumber change corresponding to FIGS. 8A and 8B. Since the tap numberdoes not cut out the spatial distribution of the PSF, a desired effectcan be obtained by a subsequent deterioration correction of the objectimage using the image restoration filter.

FIG. 10 is a conceptual diagram of the image restoration filter in thepresent embodiment. The image restoration filter is two-dimensional dataof the real space having a form illustrated in the upper part of FIG.10, and the tap number of the data is equal to the tap number of the OTFthat is information on the spatial frequency. FIG. 11 illustrates objectimages corrected by the image restoration filter. FIG. 11 is one exampleof the object images corrected by the image restoration filter made fromthe coefficient reconstruction OTF illustrated in FIGS. 8A and 8B. Ascan be seen from FIG. 11, the object images deteriorated by the imagepickup lens 112 are corrected with high resolution and with highquality.

According to the present embodiment, an image processing apparatus thatreduces an amount of storage of data required for reconstructing anoptical transfer function of an image pickup optical system can beprovided.

Embodiment 2

Next, referring to FIG. 12, Embodiment 2 of the present invention willbe described. FIG. 12 is a configuration diagram of an image pickupapparatus in the present embodiment. In Embodiment 1, the imageprocessing apparatus is used by installing a program which is configuredso that an apparatus different from the image pickup apparatus (forexample, a computer) executes the image processing method, and on theother hand in the present embodiment, the image processing apparatus isincluded in the image pickup apparatus.

An image pickup optical system 401 (a lens) including a stop 401 a and afocus lens 401 b are interchangeably attached to an image pickupapparatus 400. However, the present embodiment is not limited to this,and can also be applied to an image pickup apparatus where the imagepickup optical system is integrated. An object image obtained via theimage pickup optical system 401 is converted into an analog signal by aphotoelectric conversion performed by an image pickup element 402, andthen the analog signal is converted into a digital signal by an A/Dconverter 403. An image processor 404 performs a predetermined imageprocessing for the digital signal using each of information of a statedetector 407 and a storage portion 408. A system controller 410 controlseach of the image processor 404, a display unit 405, an image pickupoptical system controller 406, the state detector 407, and an imagerecording medium 409. The image pickup optical system controller 406controls the operation of the image pickup optical system 401, and thestate detector 407 detects the state of the image pickup optical system401 based on the information of the image pickup optical systemcontroller 406.

The image processor 404 includes the image processing apparatus in thepresent embodiment, and has a function that reconstructs the opticaltransfer function (OTF) of the image pickup optical system 401 tocorrect the object image. The storage portion 408 stores the informationrelating to the coefficient (the coefficient data), the tap number, thelens ID, the imaging condition, and the Nyquist frequency for making thereconstruction OTF calculated by the coefficient calculating apparatus.The image processing apparatus 404 matches the information obtained fromthe image pickup optical system 401 and the image pickup element 402 tothe information stored in the storage portion 408 to read theinformation required for making the image restoration filter to make theimage restoration filter. The image processor 404 also corrects theobject image recorded in the image recording medium 409 using thegenerated image restoration filter. The corrected object image isdisplayed on the display unit 405. The details of the image processingmethod in the present embodiment are omitted since they are similar toEmbodiment 1. According to the present embodiment, an image pickupapparatus (an image processing apparatus) that reduces an amount ofstorage of data required for reconstructing an optical transfer functionof an image pickup optical system can be provided.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2010-216320, filed on Sep. 28, 2010, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: a datastoring portion configured to store coefficient data for reconstructingan optical transfer function of an image pickup optical system inaccordance with a type of the image pickup optical system and an imagingcondition; a tap number determining portion configured to determine atap number of the optical transfer function that is reconstructed byusing the coefficient data in accordance with a pixel pitch of an imagepickup element; and a reconstruction portion configured to reconstructthe optical transfer function in accordance with Nyquist frequency ofthe image pickup element and the tap number in a frequency space.
 2. Theimage processing apparatus according to claim 1, wherein the tap numberdetermining portion determines the tap number of the optical transferfunction in accordance with a size of a point-spread function of eachimage height.
 3. The image processing apparatus according to claim 1,further comprising a coefficient calculating portion configured toperform a fitting processing for the optical transfer function using apredetermined function to calculate the coefficient data.
 4. The imageprocessing apparatus according to claim 3, wherein the coefficientcalculating portion performs the fitting processing for an area wherethe optical transfer function has a symmetry for each of a real part andan imaginary part, and wherein the reconstruction portion reconstructsthe optical transfer function of whole of a defined area from theoptical transfer function of the area using the symmetry of the opticaltransfer function.
 5. The image processing apparatus according to claim3, further comprising an order determining portion configured todetermine an order of the coefficient data in accordance with anaccuracy of the fitting processing, wherein the predetermined functionis an orthogonal function.
 6. The image processing apparatus accordingto claim 1, wherein the reconstruction portion selects the coefficientdata based on an ID number that specifies the image pickup opticalsystem and the imaging condition obtained from an image pickup apparatusincluding the image pickup element.
 7. The image processing apparatusaccording to claim 1, further comprising a filter processing portionconfigured to make an image restoration filter for correcting an objectimage based on the reconstructed optical transfer function to perform afilter processing for the object image on a real space using the imagerestoration filter.
 8. The image pickup apparatus comprising: an imagepickup element; a data storing portion configured to store coefficientdata for reconstructing an optical transfer function of an image pickupoptical system in accordance with a type of the image pickup opticalsystem and an imaging condition; a tap number determining portionconfigured to determine a tap number of the optical transfer functionthat is reconstructed by using the coefficient data in accordance with asize of a point-spread function of each image height and pixel pitch ofan image pickup element; and a reconstruction portion configured toperform a sampling by the tap number up to Nyquist frequency of theimage pickup element in a frequency space to reconstruct the opticaltransfer function.
 9. An image processing method comprising the stepsof: selecting coefficient data for reconstructing an optical transferfunction of the image pickup optical system in accordance with a type ofthe image pickup optical system and an imaging condition; determining atap number of the optical transfer function that is reconstructed byusing the coefficient data in accordance with a size of a point-spreadfunction of each image height and a pixel pitch of an image pickupelement; reconstructing the optical transfer function by performing asampling by the tap number up to Nyquist frequency of the image pickupelement in a frequency space; making an image restoration filter forcorrecting an object image based on the reconstructed optical transferfunction; and performing a filter processing for the object image on areal space using the image restoration filter.
 10. A non-transitorycomputer-readable storage medium storing a computer-executable program,the program comprising: selecting instructions configured to selectcoefficient data for reconstructing an optical transfer function of theimage pickup optical system in accordance with a type of the imagepickup optical system and an imaging condition; determining instructionsconfigured to determine a tap number of the optical transfer functionthat is reconstructed by using the coefficient data in accordance with asize of a point-spread function of each image height and a pixel pitchof an image pickup element; reconstructing instructions configured toreconstruct the optical transfer function by performing a sampling bythe tap number up to Nyquist frequency of the image pickup element in afrequency space; making instructions configured to make an imagerestoration filter for correcting an object image based on thereconstructed optical transfer function; and performing instructionsconfigured to perform a filter processing for the object image on a realspace using the image restoration filter.